{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/finetuning/rerankers/cohereai_reranker_finetuning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom Cohere Reranker"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook provides a tutorial on building the Cohere Custom Re-ranker using LlamaIndex abstractions. Upon completion, you'll be able to create a Custom re-ranker and utilize it for enhanced data retrieval.\n",
    "\n",
    "**Important:** This notebook offers a guide for Cohere Custom Re-ranker. The results presented at the end of this tutorial are unique to the chosen dataset and parameters. We suggest experimenting with your dataset and various parameters before deciding to incorporate it into your RAG pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's install the necessary packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-postprocessor-cohere-rerank\n",
    "%pip install llama-index-llms-openai\n",
    "%pip install llama-index-finetuning\n",
    "%pip install llama-index-embeddings-cohere"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index cohere pypdf"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize the api keys.\n",
    "\n",
    "OpenAI - For creating synthetic dataset.\n",
    "\n",
    "CohereAI - For training custom reranker and evaluating with base reranker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "openai_api_key = \"YOUR OPENAI API KEY\"\n",
    "cohere_api_key = \"YOUR COHEREAI API KEY\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n",
    "os.environ[\"COHERE_API_KEY\"] = cohere_api_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader\n",
    "from llama_index.core.node_parser import SimpleNodeParser\n",
    "\n",
    "# LLM\n",
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "# Embeddings\n",
    "from llama_index.embeddings.cohere import CohereEmbedding\n",
    "\n",
    "# Retrievers\n",
    "from llama_index.core.retrievers import BaseRetriever, VectorIndexRetriever\n",
    "\n",
    "# Rerankers\n",
    "from llama_index.core import QueryBundle\n",
    "from llama_index.core.indices.query.schema import QueryType\n",
    "from llama_index.core.schema import NodeWithScore\n",
    "from llama_index.postprocessor.cohere_rerank import CohereRerank\n",
    "from llama_index.core.evaluation import EmbeddingQAFinetuneDataset\n",
    "from llama_index.finetuning import generate_cohere_reranker_finetuning_dataset\n",
    "\n",
    "# Evaluator\n",
    "from llama_index.core.evaluation import generate_question_context_pairs\n",
    "from llama_index.core.evaluation import RetrieverEvaluator\n",
    "\n",
    "# Finetuner\n",
    "from llama_index.finetuning import CohereRerankerFinetuneEngine\n",
    "\n",
    "\n",
    "from typing import List\n",
    "import pandas as pd\n",
    "\n",
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download data\n",
    "\n",
    "We will use Lyft 2021 10K SEC Filings for training and Uber 2021 10K SEC Filings for evaluating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p 'data/10k/'\n",
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/10k/uber_2021.pdf' -O 'data/10k/uber_2021.pdf'\n",
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/10k/lyft_2021.pdf' -O 'data/10k/lyft_2021.pdf'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lyft_docs = SimpleDirectoryReader(\n",
    "    input_files=[\"./data/10k/lyft_2021.pdf\"]\n",
    ").load_data()\n",
    "uber_docs = SimpleDirectoryReader(\n",
    "    input_files=[\"./data/10k/uber_2021.pdf\"]\n",
    ").load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Curation\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Nodes.\n",
    "\n",
    "The documentation mentions that Query + Relevant Passage/ Query + Hard Negatives should be less than 510 tokens. To accomidate that we limit chunk_size to 400 tokens. (Each chunk will eventually be treated as a Relevant Passage/ Hard Negative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Limit chunk size to 400\n",
    "node_parser = SimpleNodeParser.from_defaults(chunk_size=400)\n",
    "\n",
    "# Create nodes\n",
    "lyft_nodes = node_parser.get_nodes_from_documents(lyft_docs)\n",
    "uber_nodes = node_parser.get_nodes_from_documents(uber_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use gpt-4 to create questions from chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = OpenAI(temperature=0, model=\"gpt-4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prompt to generate questions from each Node/ chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt to generate questions\n",
    "qa_generate_prompt_tmpl = \"\"\"\\\n",
    "Context information is below.\n",
    "\n",
    "---------------------\n",
    "{context_str}\n",
    "---------------------\n",
    "\n",
    "Given the context information and not prior knowledge.\n",
    "generate only questions based on the below query.\n",
    "\n",
    "You are a Professor. Your task is to setup \\\n",
    "{num_questions_per_chunk} questions for an upcoming \\\n",
    "quiz/examination. The questions should be diverse in nature \\\n",
    "across the document. The questions should not contain options, not start with Q1/ Q2. \\\n",
    "Restrict the questions to the context information provided.\\\n",
    "\"\"\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training Custom Re-ranker expects minimum 256 (Query + Relevant passage) pairs with or without hard negatives for training and 64 pairs for validation. Please note that the validation is optional.\n",
    "\n",
    "**Training:** We use first 256 nodes from Lyft for creating training pairs.\n",
    "\n",
    "**Validation:** We will use next 64 nodes from Lyft for validation.\n",
    "\n",
    "**Testing:** We will use 150 nodes from Uber."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_dataset_lyft_train = generate_question_context_pairs(\n",
    "    lyft_nodes[:256],\n",
    "    llm=llm,\n",
    "    num_questions_per_chunk=1,\n",
    "    qa_generate_prompt_tmpl=qa_generate_prompt_tmpl,\n",
    ")\n",
    "\n",
    "# Save [Optional]\n",
    "qa_dataset_lyft_train.save_json(\"lyft_train_dataset.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_dataset_lyft_val = generate_question_context_pairs(\n",
    "    lyft_nodes[257:321],\n",
    "    llm=llm,\n",
    "    num_questions_per_chunk=1,\n",
    "    qa_generate_prompt_tmpl=qa_generate_prompt_tmpl,\n",
    ")\n",
    "\n",
    "# Save [Optional]\n",
    "qa_dataset_lyft_val.save_json(\"lyft_val_dataset.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_dataset_uber_val = generate_question_context_pairs(\n",
    "    uber_nodes[:150],\n",
    "    llm=llm,\n",
    "    num_questions_per_chunk=1,\n",
    "    qa_generate_prompt_tmpl=qa_generate_prompt_tmpl,\n",
    ")\n",
    "\n",
    "# Save [Optional]\n",
    "qa_dataset_uber_val.save_json(\"uber_val_dataset.json\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have compiled questions from each chunk, we will format the data according to the specifications required for training the Custom Re-ranker.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Format and Requirements\n",
    "For both training and evaluation, it currently accepts data in the format of triplets, every row should have the following\n",
    "\n",
    "**query:** this represents the question or target\n",
    "\n",
    "**relevant_passages:** this represents a list of documents or passages that contain information that answers the query. For every query there must be at least one relevant_passage\n",
    "\n",
    "**hard_negatives:** this represents chunks or passages that don't contain answer for the query. It should be notes that Hard negatives are optional but providing atleast ~5 hard negatives will lead to meaningful improvement.\n",
    "\n",
    "[Reference](https://docs.cohere.com/docs/rerank-models)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Cohere embedding model which we use it for creating Hard Negatives.\n",
    "embed_model = CohereEmbedding(\n",
    "    cohere_api_key=cohere_api_key,\n",
    "    model_name=\"embed-english-v3.0\",\n",
    "    input_type=\"search_document\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create 3 datasets.\n",
    "\n",
    "1. Dataset without hard negatives.\n",
    "2. Dataset with hard negatives selected at random.\n",
    "3. Dataset with hard negatives selected based on cosine similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train and val datasets without hard negatives.\n",
    "generate_cohere_reranker_finetuning_dataset(\n",
    "    qa_dataset_lyft_train, finetune_dataset_file_name=\"train.jsonl\"\n",
    ")\n",
    "\n",
    "generate_cohere_reranker_finetuning_dataset(\n",
    "    qa_dataset_lyft_val, finetune_dataset_file_name=\"val.jsonl\"\n",
    ")\n",
    "\n",
    "# Train and val datasets with hard negatives selected at random.\n",
    "generate_cohere_reranker_finetuning_dataset(\n",
    "    qa_dataset_lyft_train,\n",
    "    num_negatives=5,\n",
    "    hard_negatives_gen_method=\"random\",\n",
    "    finetune_dataset_file_name=\"train_5_random.jsonl\",\n",
    "    embed_model=embed_model,\n",
    ")\n",
    "\n",
    "generate_cohere_reranker_finetuning_dataset(\n",
    "    qa_dataset_lyft_val,\n",
    "    num_negatives=5,\n",
    "    hard_negatives_gen_method=\"random\",\n",
    "    finetune_dataset_file_name=\"val_5_random.jsonl\",\n",
    "    embed_model=embed_model,\n",
    ")\n",
    "\n",
    "# Train and val datasets with hard negatives selected based on cosine similarity.\n",
    "generate_cohere_reranker_finetuning_dataset(\n",
    "    qa_dataset_lyft_train,\n",
    "    num_negatives=5,\n",
    "    hard_negatives_gen_method=\"cosine_similarity\",\n",
    "    finetune_dataset_file_name=\"train_5_cosine_similarity.jsonl\",\n",
    "    embed_model=embed_model,\n",
    ")\n",
    "\n",
    "generate_cohere_reranker_finetuning_dataset(\n",
    "    qa_dataset_lyft_val,\n",
    "    num_negatives=5,\n",
    "    hard_negatives_gen_method=\"cosine_similarity\",\n",
    "    finetune_dataset_file_name=\"val_5_cosine_similarity.jsonl\",\n",
    "    embed_model=embed_model,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Custom Reranker."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our training and validation datasets ready, we're set to proceed with the training Custom re-ranker process. Be aware that this training is expected to take approximately 25 to 45 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reranker model with 0 hard negatives.\n",
    "finetune_model_no_hard_negatives = CohereRerankerFinetuneEngine(\n",
    "    train_file_name=\"train.jsonl\",\n",
    "    val_file_name=\"val.jsonl\",\n",
    "    model_name=\"lyft_reranker_0_hard_negatives\",\n",
    "    model_type=\"RERANK\",\n",
    "    base_model=\"english\",\n",
    ")\n",
    "finetune_model_no_hard_negatives.finetune()\n",
    "\n",
    "# Reranker model with 5 hard negatives selected at random\n",
    "finetune_model_random_hard_negatives = CohereRerankerFinetuneEngine(\n",
    "    train_file_name=\"train_5_random.jsonl\",\n",
    "    val_file_name=\"val_5_random.jsonl\",\n",
    "    model_name=\"lyft_reranker_5_random_hard_negatives\",\n",
    "    model_type=\"RERANK\",\n",
    "    base_model=\"english\",\n",
    ")\n",
    "finetune_model_random_hard_negatives.finetune()\n",
    "\n",
    "# Reranker model with 5 hard negatives selected based on cosine similarity\n",
    "finetune_model_cosine_hard_negatives = CohereRerankerFinetuneEngine(\n",
    "    train_file_name=\"train_5_cosine_similarity.jsonl\",\n",
    "    val_file_name=\"val_5_cosine_similarity.jsonl\",\n",
    "    model_name=\"lyft_reranker_5_cosine_hard_negatives\",\n",
    "    model_type=\"RERANK\",\n",
    "    base_model=\"english\",\n",
    ")\n",
    "finetune_model_cosine_hard_negatives.finetune()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the jobs are submitted, you can check the training status in the `models` section of dashboard at https://dashboard.cohere.com/models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You then need to get the model id for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reranker_base = CohereRerank(top_n=5)\n",
    "reranker_model_0 = finetune_model_no_hard_negatives.get_finetuned_model(\n",
    "    top_n=5\n",
    ")\n",
    "reranker_model_5_random = (\n",
    "    finetune_model_random_hard_negatives.get_finetuned_model(top_n=5)\n",
    ")\n",
    "reranker_model_5_cosine = (\n",
    "    finetune_model_cosine_hard_negatives.get_finetuned_model(top_n=5)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will test out with first 150 nodes from Uber.\n",
    "\n",
    "1. Without Reranker.\n",
    "2. With Cohere Reranker. (without any training)\n",
    "3. With Custom reranker without hard negatives.\n",
    "4. With Custom reranker with hard negatives selected at random.\n",
    "5. With Custom reranker with hard negatives selected based on cosine similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "RERANKERS = {\n",
    "    \"WithoutReranker\": \"None\",\n",
    "    \"CohereRerank\": reranker_base,\n",
    "    \"CohereRerank_0\": reranker_model_0,\n",
    "    \"CohereRerank_5_random\": reranker_model_5_random,\n",
    "    \"CohereRerank_5_cosine\": reranker_model_5_cosine,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function to display the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_results(embedding_name, reranker_name, eval_results):\n",
    "    \"\"\"Display results from evaluate.\"\"\"\n",
    "\n",
    "    metric_dicts = []\n",
    "    for eval_result in eval_results:\n",
    "        metric_dict = eval_result.metric_vals_dict\n",
    "        metric_dicts.append(metric_dict)\n",
    "\n",
    "    full_df = pd.DataFrame(metric_dicts)\n",
    "\n",
    "    hit_rate = full_df[\"hit_rate\"].mean()\n",
    "    mrr = full_df[\"mrr\"].mean()\n",
    "\n",
    "    metric_df = pd.DataFrame(\n",
    "        {\n",
    "            \"Embedding\": [embedding_name],\n",
    "            \"Reranker\": [reranker_name],\n",
    "            \"hit_rate\": [hit_rate],\n",
    "            \"mrr\": [mrr],\n",
    "        }\n",
    "    )\n",
    "\n",
    "    return metric_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Cohere embedding model, `input_type` is different for indexing and retrieval.\n",
    "index_embed_model = CohereEmbedding(\n",
    "    cohere_api_key=cohere_api_key,\n",
    "    model_name=\"embed-english-v3.0\",\n",
    "    input_type=\"search_document\",\n",
    ")\n",
    "\n",
    "query_embed_model = CohereEmbedding(\n",
    "    cohere_api_key=cohere_api_key,\n",
    "    model_name=\"embed-english-v3.0\",\n",
    "    input_type=\"search_query\",\n",
    ")\n",
    "\n",
    "\n",
    "vector_index = VectorStoreIndex(\n",
    "    uber_nodes[:150],\n",
    "    embed_model=index_embed_model,\n",
    ")\n",
    "vector_retriever = VectorIndexRetriever(\n",
    "    index=vector_index,\n",
    "    similarity_top_k=10,\n",
    "    embed_model=query_embed_model,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_df = pd.DataFrame()\n",
    "\n",
    "embed_name = \"CohereEmbedding\"\n",
    "\n",
    "# Loop over rerankers\n",
    "for rerank_name, reranker in RERANKERS.items():\n",
    "    print(f\"Running Evaluation for Reranker: {rerank_name}\")\n",
    "\n",
    "    # Define Retriever\n",
    "    class CustomRetriever(BaseRetriever):\n",
    "        \"\"\"Custom retriever that performs both Vector search and Knowledge Graph search\"\"\"\n",
    "\n",
    "        def __init__(\n",
    "            self,\n",
    "            vector_retriever: VectorIndexRetriever,\n",
    "        ) -> None:\n",
    "            \"\"\"Init params.\"\"\"\n",
    "\n",
    "            self._vector_retriever = vector_retriever\n",
    "            super().__init__()\n",
    "\n",
    "        def _retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:\n",
    "            \"\"\"Retrieve nodes given query.\"\"\"\n",
    "\n",
    "            retrieved_nodes = self._vector_retriever.retrieve(query_bundle)\n",
    "\n",
    "            if reranker != \"None\":\n",
    "                retrieved_nodes = reranker.postprocess_nodes(\n",
    "                    retrieved_nodes, query_bundle\n",
    "                )\n",
    "            else:\n",
    "                retrieved_nodes = retrieved_nodes[:5]\n",
    "\n",
    "            return retrieved_nodes\n",
    "\n",
    "        async def _aretrieve(\n",
    "            self, query_bundle: QueryBundle\n",
    "        ) -> List[NodeWithScore]:\n",
    "            \"\"\"Asynchronously retrieve nodes given query.\n",
    "\n",
    "            Implemented by the user.\n",
    "\n",
    "            \"\"\"\n",
    "            return self._retrieve(query_bundle)\n",
    "\n",
    "        async def aretrieve(\n",
    "            self, str_or_query_bundle: QueryType\n",
    "        ) -> List[NodeWithScore]:\n",
    "            if isinstance(str_or_query_bundle, str):\n",
    "                str_or_query_bundle = QueryBundle(str_or_query_bundle)\n",
    "            return await self._aretrieve(str_or_query_bundle)\n",
    "\n",
    "    custom_retriever = CustomRetriever(vector_retriever)\n",
    "\n",
    "    retriever_evaluator = RetrieverEvaluator.from_metric_names(\n",
    "        [\"mrr\", \"hit_rate\"], retriever=custom_retriever\n",
    "    )\n",
    "    eval_results = await retriever_evaluator.aevaluate_dataset(\n",
    "        qa_dataset_uber_val\n",
    "    )\n",
    "\n",
    "    current_df = display_results(embed_name, rerank_name, eval_results)\n",
    "    results_df = pd.concat([results_df, current_df], ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check Results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(results_df)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Cohere Custom Re-ranker has led to improvements. It's important to highlight that determining the optimal number of hard negatives and whether to use random or cosine sampling should be based on experimental results. This guide presents a framework to enhance retrieval systems with Custom Cohere re-ranker.\n",
    "\n",
    "**There is potential for enhancement in the selection of hard negatives; contributions in this area are welcome from the community.**"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  },
  "vscode": {
   "interpreter": {
    "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
